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Introduction 


This  project  aims  to  demonstrate  the  feasibility  of  miniature,  inexpensive,  in  vivo  robots  to 
provide  basic  diagnosis  and  triage  in  military  environments.  This  work  comprises  the  first  phase 
of  a  two  phase  project;  the  first  phase  focuses  on  the  design  and  construction  of  an  in  vivo 
camera  robot.  The  robot  will  be  designed  to  be  fully  inserted  into  a  patient  and,  in  a  future 
project,  tested  in  patients.  The  robot  will  return  live  in  vivo  video  images  that  allow  the  surgeon 
to  explore,  diagnose  and  stabilize  the  patient  while  geographically  separated.  The  second 
phase  of  this  project  focuses  on  continued  animal  trials  as  well  as  human  testing  and  regulatory 
approval.  Our  long-term  objective  is  to  create  a  group  of  in  vivo  robots  that  can  provide 
diagnosis  and  therapeutics  at  all  echelons  of  military  medical  care. 

Body 


Task  1 :  Development  of  a  small  in  vivo  vision  system 

Planar  Manipulator  Robot 

Work  has  been  directed  towards  the  automation  of  surgical  tasks  using  the  planar  manipulator 
robot  [1],  Such  tasks  could  be  useful  in  situations  where  the  patient  is  in  a  location  far  from  a 
trained  surgeon.  A  surgeon  at  a  remote  location  could  control  the  robot  even  if  the 
communication  link  between  the  surgeon  and  the  patient  is  of  low  bandwidth  or  has  high 
latency.  The  robotic  system  has  three  main  components  including  a  visual  tracker,  controller, 
and  stereovision.  The  surgeon  is  presented  with  a  video  capture  from  the  robot  and  then  selects 
a  point  on  the  image  for  the  robot  to  move  to,  such  as  a  piece  of  tissue  to  grasp  or  cut.  Using  a 
stereo  correspondence  algorithm,  the  location  of  the  point  is  computed  in  3D  space.  The  user 
then  verifies  the  computed  point,  and  the  controller  moves  the  end  effector  to  the  desired 
position.  This  method  has  been  demonstrated  in  several  benchtop  tests  using  the  robotic 
system.  In  these  tests,  a  piece  of  rubber  band  was  placed  in  a  mount  in  the  middle  of  the  robot’s 
workspace  to  simulate  a  piece  of  tissue. 

Full  Mobility  Manipulator  Robot 

The  primary  challenge  with  the  design  of  a  full  mobility  robot  is  meeting  the  competing  design 
constraints  of  speed,  size,  and  force.  For  the  initial  prototypes  of  the  complete  robot,  the  speed 
and  force  constraints  were  met  at  the  expense  of  size.  The  first  full  mobility  robot,  NB2.0,  was 
designed  and  built  [2],  The  complete  robot  platform  includes  an  in  vivo  robot  and  a  remote 
surgeon  interface  console.  The  robot  design  consists  of  a  left  grasper  and  a  right  cautery 
forearm,  each  connected  to  a  central  body  at  a  shoulder  joint  link.  For  this  robot,  the  yaw  and 
pitch  degrees  of  freedom  are  located  at  the  shoulder  joints,  and  the  roll  and  translation  motions 
are  part  of  the  prismatic  elbow  joint.  The  body  of  the  robot  is  fitted  with  a  collar  that  is  used  with 
an  external  support  assembly  for  fixation  and  gross  positioning  of  the  robot.  For  this  prototype,  a 
standard  laparoscope  is  mounted  to  the  support  shaft  to  provide  lighting  and  visualization.  The 
surgeon  control  interface  is  located  remotely  within  the  operating  room  and  consists  of  two 
controllers,  a  video  display,  and  a  foot  pedal,  as  shown  in  Figure  1 .  This  robot  has  been  used  to 
perform  a  cholecystectomy  in  a  non-survival  animal  model  procedure.  For  this  procedure,  a 
large  incision  was  used  for  insertion  of  the  robot  due  to  the  size  of  this  initial  prototype.  The 
improved  dexterity  and  speed  of  this  robot  better  enabled  tissue  dissection.  This  robot  has  also 
been  used  in  cooperation  with  the  Compact  Bevel-geared  Robot  for  Advanced  Surgery 
(CoBRASurge)  to  perform  surgical  tasks  in  a  non-survival  animal  model  [3,4], 
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Figure  1 .  Surgeon  interface  for  manipulator  robot. 


Additional  prototypes  have  been  designed  and  built.  The  design  of  these  robots  is  similar  to  the 
previously  discussed  full  mobility  manipulator  robot,  but  with  a  one  degree-of-freedom  rotational 
elbow  joint  instead  of  a  prismatic  joint,  as  shown  in  Figure  2.  These  changes  have  increased  the 
dexterous  workspace  of  the  robot,  while  also  maintaining  sufficient  strength  and  speed.  The 
forearms  of  this  robot  were  also  designed  to  enable  the  end  effectors  to  be  interchanged 
depending  on  the  particular  tasks  being  performed.  For  example,  during  a  cholecystectomy,  a 
cautery  and  a  grasper  forearm  are  used.  Then  to  perform  knot  tying,  the  robot  could  be 
removed  and  the  cautery  end  effector  replaced  with  a  grasper.  These  robots  have  also  been 
used  in  animal  model  studies  to  successfully  perform  two  cholecystectomies.  Similar  to  the 
initial  prototype  robot,  these  robots  were  also  supported  using  an  external  assembly  mounted  to 
the  surgery  table,  and  the  robot  was  inserted  through  a  large  abdominal  incision.  Continuing 
improvements  to  this  robotic  platform  are  focused  on  reducing  the  size  of  the  robot, 
incorporating  on-board  lighting  and  cameras,  and  introducing  an  additional  degree  of  freedom  at 
the  wrist  of  the  robot. 


Figure  2.  Dexterous  manipulator  robot  (NB2.0)  with  rotational  elbow  joint. 


For  more  recent  multi-functional  robots,  the  kinematics  were  modified  to  allow  for  a  greater 
dexterous  workspace,  as  in  the  NB2.1  [5].  This  robotic  platform  is  designed  specifically  for 
Laparoendoscopic  Single-Site  Surgery  (LESS),  and  consists  of  a  multi-functional  robot  and  a 
remote  surgeon  interface.  This  robot  is  designed  to  be  inserted  through  a  single  incision  and  be 
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contained  completely  within  the  peritoneal  cavity.  The  miniature  dexterous  in  vivo  robot,  shown 
in  Figure  3,  consists  of  two  arms  connected  to  a  main  body  segment.  The  main  body  of  the 
robot  is  composed  of  three  modules.  These  modules  can  be  independently  inserted  through  a 
single  incision  and  then  assembled  once  inside  the  peritoneal  cavity  to  provide  surgical 
capabilities.  Following  assembly  of  the  robot,  a  mounting  rod  is  introduced  through  the  insertion 
incision  and  mated  to  the  center  module  to  support  the  robot  within  the  peritoneal  cavity.  The 
mounting  rod  is  supported  by  an  external  support  system  that  is  mounted  to  the  rails  of  the 
operating  table.  Gross  positioning  of  the  robot  within  the  peritoneal  cavity  can  be  accomplished 
by  adjusting  the  depth  and  angle  of  the  support  rod. 


Figure  3.  Miniature  in  vivo  robot  (NB2.1)  performing  a  cholecystectomy. 


Each  outer  module  of  the  body,  shown  in  Figure  4,  is  connected  to  an  arm  at  a  two-degree  of 
freedom  joint.  The  shoulder  joint  links  consist  of  a  distal  joint  providing  yaw,  and  a  proximal  joint 
providing  pitch.  Each  arm  consists  of  a  two-degree  of  freedom  rotational  elbow  joint. 
Specialized  end  effectors  on  each  forearm  can  be  interchanged  to  provide  tissue  manipulation, 
monopolar  cautery,  and  intracorporeal  suturing  capabilities.  Each  outer  module  is  connected  to 
a  center  module  that  contains  two  cameras.  These  cameras  can  provide  a  stereoscopic 
visualization  with  panning  and  tilting.  An  ultra  bright  LED  is  also  contained  in  the  center  module 
to  provide  on-board  lighting.  The  robot  joints  are  independently  controlled  using  a  proportional- 
integral-derivative  (PID)  control  method,  with  actuation  provided  by  coreless  permanent  magnet 
direct  current  motors  with  magnetic  encoders.  These  motors  are  housed  within  the  arms  and 
body  of  the  robot. 

The  multi-functional  robot  platform  has  been  prototyped  tested  in  four  non-survival 
cholecystectomies  in  a  porcine  model  at  the  University  of  Nebraska  Medical  Center.  All 
experimental  protocols  were  approved  by  the  Institutional  Animal  Care  and  Use  Committee 
(IACUC).  The  robot  was  supported  above  the  animal  using  the  external  support  assembly 
described  previously.  For  these  procedures,  a  large  transabdominal  incision  was  made  to 
provide  access  to  the  peritoneal  cavity  due  to  the  size  of  the  robot.  The  robot  was  then 
positioned  within  the  proper  workspace  for  performing  a  cholecystectomy.  The  surgeon 
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controlled  the  robot  from  the  control  console  located  remotely  within  the  operating  room.  The 
procedure  was  then  performed  similarly  to  a  standard  laparoscopic  cholecystectomy.  The 
grasper  end  effector  was  extended  to  grasp  the  cystic  duct  and  lifted  while  the  cautery  end 
effector  performed  tissue  dissection.  This  stretch  and  dissect  task  was  performed  iteratively 
until  a  full  cholecystectomy  was  completed.  Stapling  of  the  cystic  duct  and  supplementary 
retraction  were  performed  using  standard  laparoscopic  tools. 


Figure  4.  Separated  robot  outer  arm  modules. 


Currently,  improvements  are  being  made  to  the  NB2.1  robot  design  to  improve  performance 
without  changing  the  kinematics  of  the  robot.  First,  heat  sinks  and  thermocouples  are  being 
added  to  the  motors  housed  in  the  robot.  These  thermocouples  will  allow  for  temperature 
monitoring  of  the  motors,  allowing  for  shutoff  before  permanent  motor  damage  occurs.  Further, 
the  housing  of  the  motors  are  being  changed  to  a  “clamshell”  design  to  eliminate  the  twisting 
required  for  the  current  assembly  method  of  the  robot.  This  method  should  provide  a  tighter 
motor  fit  and  decrease  the  potential  for  twisting  or  disconnection  of  the  ribbon  cables  to  the 
motors.  Further,  the  metal  gears  in  the  gripper  attachment  are  being  changed  to  plastic  to 
electrically  isolate  the  gripper  from  the  electronics  of  the  robot.  This  should  help  reduce  the 
potential  for  the  cautery  tools  to  interfere  with  the  manipulation  of  the  robot  through  direct 
conduction  with  the  robot  grippers. 


Control  Interface 


A  new  control  and  interface  system  was  developed  for  the  full  mobility  manipulator  robots.  The 
major  changes  between  the  old  system  and  the  new  system  are  the  changes  in  the  hand-held 
controllers,  PID  control  algorithms,  programming,  and  user  interface.  The  surgeon  now  controls 
the  robot  using  two  Phantom  Omni  controllers  (SensAble)  as  shown  in  Figure  5.  The 
improvement  over  the  old  controllers  is  that  these  hand-held  haptic  controllers  allow  the  surgeon 
to  feel  the  limits  of  the  workspace  and  to  feel  collisions  between  the  two  robotic  arms.  This 
allows  the  surgeon  to  be  better  immersed  into  the  virtual  surgical  field.  Additionally,  the 
controllers  can  be  locked  into  place  anywhere  in  their  workspace.  This  means  that  the  surgeon 
can  lock  the  controllers  and  walk  away  and  when  the  he  returns  the  controllers  will  be  in  the 
same  place  as  when  he  left. 
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The  PID  control  algorithms  that  control  the  robot  motors  have  been  updated  to  include  both 
position  and  current  feedback.  The  addition  of  current  feedback  protects  the  motors  from  being 
overdriven  to  the  point  of  failure.  By  reading  the  amount  of  current  going  through  the  motor  and 
limiting  it,  the  motor  life  can  be  protected.  Further,  an  entirely  new  control  program  was 
developed  to  address  several  concerns  from  the  old  program.  Simultaneously,  the  graphical 
user  interface  for  the  control  program  was  updated  and  is  shown  in  Figure  6.  These  changes 
allow  the  program  to  run  faster,  be  modified  more  easily,  and  to  communicate  with  the  user 
more  efficiently. 


Figure  5.  To  control  the  robot,  the  surgeon  manipulates  two  Phantom  Omni  haptic  controllers. 
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Figure  6.  Updated  Graphical  User  Interface  (GUI)  for  the  full  mobility  robots. 
Stereoscopic  Visualization 

A  stereoscopic  visualization  system  is  being  developed  for  the  robot  [6],  The  system  consists  of 
two  main  subsystems.  The  first  subsystem  consists  of  the  two  CMOS  cameras  that  are  mounted 
to  central  body  of  the  in  vivo  miniature  robot.  The  cameras  can  be  panned  and  tilted  to  provide 
an  adjustable  viewpoint  throughout  a  surgical  procedure.  Each  cameras  feeds  live  video  to  the 
monitors,  where  the  video  is  reflected  through  a  series  of  mirrors  to  surgeon.  The  screens  and 
the  mirrors  make  up  the  stereoscopic  display  subsystem,  and  are  housed  inside  of  a 
stereoscopic  box.  The  combination  of  the  placement  of  the  cameras  and  mirrors  allows  for 
stereoscopic  vision. 

The  cameras  are  placed  and  angled  to  give  depth  perception  comparable  to  what  a  human 
would  have  focusing  on  an  object  from  one  meter  away.  Humans  have  depth  perception 
because  our  eyes  are  approximately  64  millimeters  away  from  one  another,  this  displacement 
allows  our  eyes  to  see  two  different  images,  which  our  brain  then  processes  and  combines  into 
one  image.  The  cameras  work  in  the  same  way.  Based  on  the  limiting  dimensions  of  the 
workspace,  the  cameras  are  placed  approximately  12.9  millimeters  apart  and  angled  inward  at  a 
2.1  degree  angle.  These  values  were  found  using  proportionality  laws  and  similar  triangles. 

The  stereoscopic  display  subsystem  is  shown  in  Figure  7.  The  two  monitors  are  placed 
vertically  at  the  rear  of  the  rectangular  box,  providing  live  video  from  the  left  and  right  cameras, 
respectively.  A  set  of  mirrors  is  used  to  feed  the  live  video  from  each  monitor  to  the  surgeon. 
One  mirror  is  placed  directly  in  front  of  each  monitor  in  the  front  corners  of  the  stereoscopic 
system  at  a  45  degree  angle  to  the  monitors.  The  second,  smaller  mirror,  is  placed  in  the  center 
of  the  system  and  parallel  to  the  larger  set  of  mirrors.  Two  sets  of  mirrors  are  necessary  to  flip 
the  image  twice,  so  the  image  that  is  processed  by  the  brain  is  the  original  image  and  not  a 
reflection. 
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These  mirrors  can  sit  at  any  angle  to  the  screens  as  long  as  they  remain  parallel  to  each  other, 
if  they  do  not  remain  parallel  to  each  other  the  viewer  will  encounter  keystone  distortion,  this 
type  of  distortion  occurs  when  the  image  is  stretched  resulting  in  one  side  appearing  larger  and 
the  other  appearing  smaller  [25].  This  distortion  is  exaggerated  because  it  occurs  in  both  the 
right  view  and  the  left  view.  This  distortion  of  the  images  reduces  the  acuity  and  quality  of  the 
images,  so  it  is  essential  that  they  remain  parallel  in  this  type  of  stereoscopic  viewer.  To 
compensate  for  any  level  of  dissimilar  angles,  levels  of  adjustability  are  placed  on  all  of  the 
mirrors.  Three  adjustable  screws  sit  behind  each  of  the  mirrors,  allowing  the  user  to  correct  for 
distortion  in  the  image. 

The  smaller  set  of  mirrors  are  placed  adjacent  to  one  another  so  the  image  can  be  centered  on 
the  mirrors  at  the  same  distance  apart  as  the  human  eyes.  The  viewer  places  their  eyes  in  line 
with  the  smaller  set  of  mirrors  of  focuses  on  the  image.  The  left  camera  feeds  the  video 
information  through  the  left  set  of  mirrors  and  this  information  in  processed  by  the  left  eye,  while 
the  right  cameras  feeds  the  video  through  the  right  set  of  mirrors  and  is  processed  by  the  right 
eye.  The  brain  then  processes  the  different  images  from  the  two  cameras  and  to  create  one 
image,  providing  the  viewer  with  the  perception  of  depth. 


The  stereoscopic  visualization  system  with  haptic  technology  was  successfully  used  in  non¬ 
survival  animal  procedures.  Surgeons  successfully  used  the  entire  system  to  complete  the  task 
of  suturing.  The  system  accurately  gave  the  surgeon  perception  of  depth.  The  surgeon  noted 
that  the  stereoscopic  visualization  system  was  of  significant  benefit  in  this  situation.  While  using 
this  robot,  it  helped  the  surgeon  accurately  position  the  end  effectors.  This  system  will  be  used 
in  more  complex  procedures  in  the  near  future. 


Modular  Wireless  Mobile  Robot 


Circuit  boards  and  cautery  have  been  successfully  tested  for  the  modular  wireless  mobile  robot 
platform.  The  robot  design  is  being  adjusted  to  fit  the  components  into  the  robot’s  modular 
payload  housing.  Current  research  is  also  focused  on  evaluating  the  staple/clamp  arm  and  its 
ability  to  provide  enough  pressure  to  stop  blood  flowing  through  a  vessel.  These  evaluations 
using  Finite  Element  Software  are  being  compared  with  the  previously  completed  ex  vivo  and  in 
vivo  test  results  of  the  biopsy  grasper.  The  tissue  interaction  during  a  liver  biopsy  is  also  being 
investigated.  Work  is  also  being  done  to  create  a  soft-tissue  model  to  evaluate  the  performance 
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of  end  effectors  such  as  the  biopsy  grasper.  Other  payload  variations  for  surgical  task 
assistance  are  in  the  conceptual  stages. 

Task  2:  Development  of  an  “easy  to  carry”  relay  system  and  remote  user  interface 

Wireless  control  and  video  capabilities  are  being  developed  for  integration  with  the  manipulator 
robots.  The  control  design  allows  the  robot  to  be  controlled  through  either  a  graphic  user 
interface,  or  remotely  with  the  previously  described  surgeon  console  using  existing  wired  and 
wireless  local  Ethernet  network.  An  overview  of  the  combined  system  for  controlling  the  in  vivo 
robot  remotely  is  shown  in  Figure  8. 


Controller  Side 


Robot  Side 


Laptop  computer 

Surgeon 
control  consol 


Figure  8.  Overview  of  remote  user  interface  system. 

The  system  software  uses  a  layered  design.  The  main  benefit  of  a  layered  software  design  is 
the  isolation  of  the  different  levels  of  control  and  communication.  Low  level  control  of  robot 
actuators  and  cameras  as  well  as  the  surgeon  control  console  are  in  the  lowest  software  layer. 
Communication,  the  graphical  user  interface,  and  user  management  is  at  a  different,  higher 
layer.  The  layers  communicate  through  messages,  which  makes  the  design  more  portable  to 
different  robots.  TCP/IP  protocol  is  used  for  the  control  messages  between  the  robot  and  the 
control  station.  The  video  from  the  robot  camera  is  streamed  through  an  embedded  plug-in  for 
VLC  player  using  asf/wmv  encoding  with  200ms  buffering. 

A  benchtop  test  of  the  remote  user  interface  was  performed  between  the  surgeon  console, 
located  at  the  University  of  Nebraska  Medical  Center  (UNMC)  in  Omaha,  Nebraska,  and  the 
robot  located  at  the  University  of  Nebraska-Lincoln  (UNL),  in  Lincoln,  Nebraska.  Results 
showed  an  average  control  delay  of  251  ms  while  performing  the  remote  control.  The  average 
local  control  delay  of  the  robot  using  a  loopback  IP  address  is  about  91  ms.  For  this  test,  the 
control  station  was  connected  through  a  wireless  802.1 1g  to  the  UNMC  wireless  network,  while 
the  robot  was  connected  through  to  a  802.2  wired  network  to  UNL’s  network. 

Difficulty  has  been  encountered  in  ensuring  quality  video  is  being  made  available  to  the 
surgeon.  A  significant  challenge  has  been  to  consistently  provide  high  quality  video  (in  terms  of 
resolution  and  frame  rates)  with  low  enough  latency  that  robot  motion  commands,  from  the 
surgeon,  correspond  spatially  and  temporally  with  the  robot’s  actual  motion.  This  latency  stems 
primarily  from  two  sources:  the  off-the-shelf  video  frame  grabber  needed  to  convert  the  analog 
image  to  digital,  and  the  video  buffer  employed  by  the  off-the-shelf  video  playback  software, 
which  buffers  frames  to  provide  smooth  playback.  Additional  manipulations,  provided  within  the 
approved  no-cost  extension,  are  necessary  to  finalize  the  user  datagram  protocols  (UDP)  for 
use  with  the  remote  user  interface  and  relay  station.  Once  the  digital  image  sensor  is  integrated 
with  the  in  vivo  robot,  the  relay  system  and  remote  user  interface  will  be  updated  to  capture  and 
transmit  the  digital  image  directly,  rather  than  using  the  frame  grabber.  This  will  eliminate  the 
analog  to  video  conversion  latency  and  help  to  reduce  latency  in  the  video  transmission, 
ultimately  making  the  system  more  surgeon  user  friendly.  If  further  latency  reduction  is  required, 
portions  of  the  playback  software  will  need  to  be  rewritten  to  reduce  buffering.  This  may  result  in 
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increased  video  jitter  under  some  network  conditions,  but  will  eliminate  the  current  frame  buffer 
latency. 

Other  work  has  focused  on  developing  a  method  of  controlling  the  video  parameters  of  the 
digital  imaging  board  which  utilizes  the  Omni  Vision  OV2640  imaging  chip  with  the  Omni  Vision 
OV550  camera  bridge  processor.  The  resulting  software  solution  enables  direct  access  to  the 
imager  board’s  memory  registers  and  supports  control  actions  such  as  changing  the  output 
format,  i.e.  choosing  image  compression  algorithm,  setting  frame  size  and  refresh  rates,  and 
enabling/disabling  the  digital  signal  processing  functions  offered  by  the  imaging  chip.  The  image 
processing  functions  include  automatic  or  manual  white  balance  control  and  the  adjustment  of 
the  image  contrast  and  brightness.  The  second  major  focus  resulted  in  the  development  of  a 
prototype  stereoscopic  vision  system  to  be  developed  and  implemented.  The  system  consists  of 
two,  2-MegaPixel  Logitech  C905  webcams,  a  polarized  ED  monitor  (Hyundai,  model  W220s) 
and  a  computer  with  the  MATLAB  environment  installed.  Frames  from  the  webcams  are 
acquired  using  MATLAB’s  Image  Acquisition  Toolbox  and  interlaced  to  produce  a  full-color 
stereoscopic  video  stream  which  allows  end-users  to  perceive  the  true  depth  of  the  observed 
scene  in  real-time. 

Task  3:  Develop  procedures  and  techniques  for  military  use  of  in  vivo  robots 

Continued  benchtop  experiments  elicited  several  potential  military  and  clinical  applications 
including  immediate  exploration,  diagnosis,  triage,  stabilizing  treatment,  and  transmission  of 
medical  information.  Field  deployable  in  vivo  robots,  with  minimal  size  and  weight,  have  the 
capability  to  positively  impact  both  forward  and  noncombatant  care  environments  through 
decreased  wound  infections,  pain,  recovery  time,  and  adhesions. 

Ongoing  testing  has  indicated  that  we  are  able  to  effectively  grasp  and  manipulate  tissues  with 
increased  dexterity,  which  has  allowed  successful  robotic  abdominal  exploration  and  blood 
vessel  ligation.  Abdominal  surgical  procedures  performed  by  the  most  recent  miniature  robotic 
prototype  have  produced  liver,  splenic,  and  visceral  artery  bleeding.  Subsequently,  we  have 
been  developing  various  robotic  devices  including  clamps,  cautery,  and  clips  to  control  blood 
loss.  Upcoming  animal  experimentation,  slated  to  occur  in  March  2010,  will  accommodate 
additional  device  manipulation  and  testing  necessary  to  mitigate  and  prevent  liver,  splenic,  and 
visceral  artery  bleeding.  Continued  bench  top  experimentation  will  implement  as  well  as 
determine  the  efficacy  of  clamps,  cautery,  and/or  clip  devices  within  the  robotic  prototype  to 
control  blood  loss. 

Task  4:  Integration  and  testing 

The  above  sub-systems  will  be  integrated  into  a  deliverable  system.  The  above  tasks  build  on 
previous  preliminary  studies  performed  by  this  investigative  team.  Large  aspects  of  Tasks  1  and 
2  have  been  demonstrated  in  clinical  environments  on  animal  models.  A  continued  challenge  of 
the  work,  to  be  performed  within  the  ongoing  no-cost  extension  period,  will  be  to  make  these 
systems  function  in  forward  situations.  Subsequently,  much  work  has  been  done  to  date 
including  the  design,  construction,  and  testing  of  more  than  five  robot  prototypes.  Additional 
time  allows  for  the  continued  development  and  testing  needed  to  incorporate  the  outcomes  of 
Tasks  1-3  above. 
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Key  Research  Accomplishments 

•  Four  full  mobility  manipulator  robots  have  been  built  and  demonstrated  in  benchtop 
testing  and  non-survival  animal  model  surgeries. 

•  PID  controllers  have  been  developed  for  the  full  mobility  manipulator  robots  allowing  for 
real-time  position  tracking  of  a  master  manipulator.  This  has  greatly  improved  the  speed 
and  dexterity  of  the  robot. 

•  Monopolar  electrosurgery  capabilities  have  been  introduced  with  the  manipulator  robots. 
This  significantly  improves  performance  compared  to  previous  heat  element  cautery 
methods. 

•  Stereovision  display  systems  have  been  developed  for  the  robot. 

•  A  stereo  camera  circuit  board  has  been  designed,  built  and  tested  for  the  manipulator 
robots. 

•  The  video  from  the  manipulator  robot  cameras  have  been  recorded  wireless  using  RF 
transceivers. 

•  Circuit  boards  for  the  modular  wireless  robot  platform  have  been  tested  and  the  platform 
design  is  being  adjusted  to  allow  for  implementation. 

•  A  graphical  user  interface  has  been  developed  to  control  the  prototype  planar 
manipulator  robots  through  the  Ethernet. 

•  The  user  interface  for  the  prototype  planar  manipulator  robots  has  been  tested  through 
wired  and  wireless  networks.  Software  has  been  developed  to  control  the  camera  and 
image  settings  from  the  manipulator  robots  through  the  Ethernet. 

•  Finite  Element  Analysis  of  the  biopsy  grasping  mechanism  for  the  wireless  mobile  robot 
is  being  performed  and  compared  with  previous  in  vivo  and  ex  vivo  test  results. 

•  A  model  to  evaluate  the  performance  of  robotic  end  effectors  (such  as  biopsy)  when 
cutting  organ  tissue  is  being  developed. 

Reportable  Outcomes 

Refereed  Journal  Publications: 

Lehman,  A.C.,  Dumped,  J.,  Wood,  N.A.,  Visty,  A.Q.,  Farritor,  S.M.,  Varnell,  B.,  Oleynikov,  D., 
“Natural  Orifice  Translumenal  Endoscopic  Surgery  with  a  Miniature  In  Vivo  Surgical  Robot 
(Video),”  Surgical  Endoscopy,  23(7),  2009. 

Canes,  D.,  Lehman,  A.C.,  Farritor,  S.M.,  Oleynikov,  D.,  Desai,  M.M.,  “The  Future  of  NOTES 
Instrumentation:  Flexible  Robotics  and  In  Vivo  Minirobots,”  Journal  of  Endourology,  23(5): 
787-92,  2009. 

Shah,  B.C.,  Buettner,  S.L.,  Lehman,  A.C.,  Farritor,  S.,  Oleynikov,  D.,  “Miniature  In  Vivo  Robotics 
and  Novel  Robotic  Surgical  Platforms,”  Urologic  Clinics  of  North  America,  36(2):  251-263,  2009. 

Lehman,  A.C.,  Tiwari,  M.M.,  Shah,  B.C.,  Farritor,  S.M.,  Nelson,  C.A.,  Oleynikov,  D.,  “Recent 
Advances  in  Robotic  Manipulators  and  Miniature  In  Vivo  Robotics  for  Minimally  Invasive 
Surgery,”  Journal  of  Mechanical  Engineering  Science,  Submitted. 

Lehman,  A.C.,  Wood,  N.A.,  Farritor,  S.M.,  Goede,  M.R.,  Oleynikov,  D.,  “Dexterous  Robot  for 
Single  Incision  Advanced  Minimally  Invasive  Surgery,”  Surgical  Endoscopy,  Submitted. 

Qadi,  A.,  Goddard,  S.,  Huang,  J.,  Farritor,  S.,  “On  Providing  Performance  Guarantees  of  an 
Autonomous  Mobile  Robot,”  International  Journal  of  Robotics  and  Automation,  Submitted. 
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Tiwari,  M.M.,  Reynoso,  J.F.,  Lehman,  A.C.,  Tsang,  A.W.,  Farritor,  S.M.,  Oleynikov,  D.,  “In  Vivo 
Miniature  Robots  for  Natural  Orifice  Surgery:  State  of  the  Art  and  Future  Perspectives,  World 
Journal  of  Gastrointestinal  Surgery,  Submitted. 

Wortman,  T.D.,  Strabala,  K.W.,  Lehman,  A.C.,  Farritor,  S.M.,  Oleynikov,  D.  “Laparoendoscopic 
Single-Site  Surgery  using  a  Multi-Functional  Miniature  In  Vivo  Robot,”  The  International  Journal 
of  Medical  Robotics  and  Computer  Assisted  Surgery,  Submitted. 

Refereed  Conference  Publications  and  Presentations: 

Zhang,  X.,  Lehman,  A.C.,  Nelson,  C.A.,  Farritor,  S.M.,  Oleynikov,  D.,  “Cooperative  Robotic 
Assistant  for  Laparoscopic  Surgery:  CoBRASurge,”  IEEE/RSJ  International  Conference  on 
Intelligent  Robots  and  Systems,  St.  Louis,  MO,  October  2009. 

Dumped,  J.,  Lehman,  A.C.,  Wood,  N.A.,  Oleynikov,  D.,  Farritor,  S.M.,  “Semi-Autonomous 
Surgical  Tasks  Using  a  Miniature  In  Vivo  Surgical  Robot,”  Engineering  in  Medicine  and  Biology 
Conference,  Minneapolis,  MN,  September  2009. 

Bornhoft,  J.M.,  Strabala,  K.W.,  Wortman,  T.D.,  Lehman,  A.C.,  Oleynikov,  D.,  Farritor,  S.M., 
“Stereoscopic  Visualization  and  Haptic  Technology  Used  to  Create  a  Virtual  Environment  for 
Remote  Surgery,”  ASME  2010  World  Conference  on  Innovative  Virtual  Reality,  Ames,  IA,  May 
2010,  Submitted. 

Book  Chapters: 

Farritor,  S.M.,  Lehman,  A.C.,  Oleynikov,  D.,  “Miniature  In  Vivo  Robots  for  NOTES,”  Surgical 
Robotics  -  Systems,  Applications,  and  Visions,  Submitted. 

Medical  Conference  Publications  and  Presentations: 

Lehman,  A.C.,  Wood,  N.A.,  Farritor,  S.M.,  Goede,  M.R.,  Oleynikov,  D.,  “Dexterous  Robot  for 
Single  Incision  Advanced  Minimally  Invasive  Surgery,”  The  Society  of  American  Gastrointestinal 
and  Endoscopic  Surgeons,  Phoenix,  AZ,  April  2009. 

Wortman,  T.D.,  Strabala,  K.W.,  Lehman,  A.C.,  Farritor,  S.M.,  Oleynikov,  D.  “Laparoendoscopic 
Single-Site  Surgery  using  a  Multi-Functional  Miniature  In  Vivo  Robot,”  Minimally  Invasive 
Robotic  Association,  San  Diego,  CA,  January  2010. 

Conclusion 

Our  long-term  goal,  to  use  image-guided  miniature  robots  to  convert  open  and  laparoscopic 
surgeries  to  the  NOTES  approach,  can  be  realized  through  the  development  of  a  family  of  in 
vivo  robots.  Completion  of  the  current  statement  of  work  is  a  critical  first  step  toward  this  effort 
as  it  builds  on  previous  successes  and  focuses  on  developing  an  image-guided  robot  capable  of 
provisions  of  basic  diagnosis  and  triage. 


Initial  testing  has  indicated  that  the  default  imager  settings,  e.g.,  hue  and  saturation,  are  not 
ideal  for  surgical  environments.  Supporting  documentation  and  application  software  from  the 
manufacturer  for  this  sensor  are  incomplete  in  order  to  determine  the  imaging  settings  that  will 
yield  optimal  video  for  surgical  applications  Subsequently,  additional  in  vivo  experimentation  is 
necessary  to  design  an  apparatus  and  protocols  that  produce  idyllic  in  vivo  imager  settings. 
Continued  development  of  an  “easy  to  carry”  relay  system  and  remote  user  interface  is 
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necessary  to  enable  the  transference  of  real-time  video  to  perform  innovative  robotic  diagnosis 
and  intervention  in  forward  environments.  Difficulties  with  regard  to  video  latency,  multicasting, 
reliability  and  congestion  control  issues  require  additional  manipulation  of  UDP  to  ensure  high 
quality  video  with  low  enough  latency  that  robot  motion  commands  from  the  surgeon  correspond 
spatially  and  temporally  with  the  in  vivo  robot’s  actual  motion.  Several  functional  prototypes 
capable  of  tissue  manipulation,  abdominal  exploration,  and  blood  vessel  ligation  have  been 
developed,  but  require  additional  device  manipulation  and  testing  to  mitigate  and  prevent  liver, 
splenic,  and  visceral  artery  bleeding.  Continued  bench  top  experimentation  will  implement  as 
well  as  determine  the  efficacy  of  clamps,  cautery,  and/or  clip  devices  within  the  robotic 
prototype  to  control  blood  loss.  Additional  time  provided  by  the  no-cost  extension  period  allows 
for  the  continued  development  and  testing  needed  to  incorporate  the  outcomes  of  Tasks  1-4 
above.  This  revolutionary  robotic  technology  has  demonstrated  its  applicability  and  benefit  in 
natural  orifice  and  single  incision  minimally  invasive  surgical  procedures.  Such  procedures  are 
virtually  impossible  to  perform  without  the  design  and  creation  of  new  tools  like  our  miniature 
robots. 

The  small,  in  vivo  robots  developed  in  this  study  may  enable  lifesaving  diagnosis  and  triage  in 
more  forward  military  environments.  The  second  phase  of  this  project  will  focus  on  continued  in 
vivo  testing  as  well  as  the  acquisition  of  regulatory  approval.  The  portability  and  survivability  of 
our  technology  in  forward,  rugged  situations  is  a  substantial  challenge  to  be  met;  we  plan 
continued  trauma  model  testing  and  prototype  development  to  meet  this  challenge.  Project 
success  will  have  a  direct  impact  on  combat  medical  care,  thus  matching  TATRC’s  Research 
Area  of  Interest  B,  Combat  Casualty  Care  Research  Program.  The  new  robotic  technology  will 
also  be  useful  in  many  other  levels  of  military  medical  care  including  level  five  treatment  and 
stateside  facilities  as  well  as  significantly  impacting  the  application  of  NOTES  and  single  incision 
laparoscopic  surgery  everywhere. 
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